Dental imaging mixed reality system

ABSTRACT

An imaging system is described. The imaging system accesses first imaging data of a specimen using a first sensor device. The first imaging data comprising volumetric data. The imaging system accesses second imaging data of the specimen using a second sensor device. The second imaging data comprising surface data. The imaging system registers a common anatomical region of the specimen in the first imaging data and the second imaging data and generates a composite image based on the registered common anatomical region. The composite image indicates the volumetric data and the surface data of the specimen.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to U.S. Provisional PatentApplication Ser. No. 62/799,938, filed Feb. 1, 2019, which is herebyincorporated by reference in its entirety.

BACKGROUND

The subject matter disclosed herein generally relates to the processingof data. Specifically, the present disclosure addresses systems andmethods for processing dental images and providing augmented informationon a composite three-dimensional model based on the dental images.

Current techniques for assessing periodontal health involves a dentistmanually measuring gum disease of a patient at different points with adental instrument. This manual procedure does not allow the dentist orthe patient to easily visualize a current state of gum disease of thepatient. Furthermore, it is difficult for the dentist to examine aregion of interest (e.g., dental caries) in the mouth of the patient andespecially, the position of an instrument (inside the mouth of thepatient) relative to the region of interest.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

To easily identify the discussion of any particular element or act, themost significant digit or digits in a reference number refer to thefigure number in which that element is first introduced.

FIG. 1 illustrates a network environment for operating an imaging systemin accordance with one example embodiment.

FIG. 2 illustrates an imaging system in accordance with one exampleembodiment.

FIG. 3 illustrates a composite image application in accordance with oneexample embodiment.

FIG. 4 illustrates a mixed reality (MR) application in accordance withone example embodiment.

FIG. 5 illustrates a dental disease MR display module in accordance withone example embodiment.

FIG. 6 illustrates a method for generating a composite image inaccordance with one example embodiment.

FIG. 7 illustrates a method for generating a 3D model in accordance withone example embodiment.

FIG. 8 illustrates a method for generating MR information in accordancewith one example embodiment.

FIG. 9 illustrates a routine 900 in accordance with one embodiment.

FIG. 10 illustrates an example operation of the composite imageapplication in accordance with one example embodiment.

FIG. 11 illustrates an example operation of the dental instrument MRdisplay module in accordance with one example embodiment.

FIG. 12 illustrates an example of combining a first and second dentalimaging data to generate a composite image in accordance with oneexample embodiment.

FIG. 13 illustrates a reference object in accordance with one exampleembodiment.

FIG. 14 illustrates an example operation of the reference object inaccordance with one example embodiment.

FIG. 15 illustrates an example of a dental instrument with inertialsensors in accordance with one example embodiment.

FIG. 16 illustrates a dental instrument with inertial sensors inaccordance with one example embodiment.

FIG. 17 illustrates a cross-section of a tooth in accordance with oneexample embodiment.

FIG. 18 illustrates a composite image of a cross-section of a tooth inaccordance with one example embodiment.

FIG. 19 illustrates an example operation of the imaging system inaccordance with one example embodiment.

FIG. 20 illustrates another example operation of the imaging system inaccordance with one example embodiment.

FIG. 21 illustrates another example operation of the imaging system inaccordance with one example embodiment.

FIG. 22 is a diagrammatic representation of a machine in the form of acomputer system within which a set of instructions may be executed forcausing the machine to perform any one or more of the methodologiesdiscussed herein, according to an example embodiment.

DETAILED DESCRIPTION

Example methods and systems are directed to a method for dental imagingwith augmented information. Examples merely typify possible variations.Unless explicitly stated otherwise, components and functions areoptional and may be combined or subdivided, and operations may vary insequence or be combined or subdivided. In the following description, forpurposes of explanation, numerous specific details are set forth toprovide a thorough understanding of example embodiments. It will beevident to one skilled in the art, however, that the present subjectmatter may be practiced without these specific details.

The present application describes a method for providing diagnosticinformation regarding a patient's oral periodontal health. Currentmethods are slow, painful, and inaccurate (e.g., a dentist pokes apatient's gum line with a little thin stick and calls out arbitrarynumbers). In one example embodiment, the information from two or moreimaging techniques (e.g., intraoral scanner and cone beam computerizedtomography) is combined and provides additional information that cannotbe determined using the individual components separately.

The present application also describes a method for virtual positioningof a dental instrument in relation to a particular “region of interest”for a dental procedure. For example, a region of interest may include alocation of cavities when drilling a filling (also can be used to locatetooth nerve when performing a filling), a location of bone for implantplacement, and a location of target nerve for tough anestheticinjections. Currently, the dentist assesses or estimates the region ofinterest based on visible external landmarks in the mouth of thepatient.

The present application describes a comprehensive assessment of aperson's periodontal health and individualized anatomical landmarksbased on multiple imaging sources and augment a clinician's clinicaltools (e.g., handpiece) to virtually interact/position itself in thesame digital workspace.

Example advantages of digital diagnosis include:

imaging can be performed by auxiliary dental staff, accurate than thetraditional methods of measuring gum diseases (3 point measurements onthe gum) which makes it easy to miss;

easier visualization by the doctor and patients (patient education andeasier to “sell” treatment);

quantity different markers of periodontal (e.g., gum) healthinstantaneously. The imaging data can be used to improve gum measurementover time (e.g., via machine learning model) and visualize healthtrends.

Example advantage of augmented reality for dental tools include:

more precise than just guessing using local landmarks;

can be used to definitively avoid problem areas (e.g., nerve chamberwhen drilling a tooth for a filling);

can be used for multiple types of dental procedures (e.g., fillings,crown preps, injections)—system can be used with interchangeable parts;

can be integrated into normal dental workflow (uses tools that dentistsare comfortable with).

In one example embodiment, the present application describes a methodcomprising: accessing first imaging data of a specimen using a firstsensor device, the first imaging data comprising volumetric data;accessing second imaging data of the specimen using a second sensordevice, the second imaging data comprising surface data; and generatinga composite image based on the first and second imaging data, thecomposite image indicating the volumetric data and the surface data ofthe specimen.

In another example embodiment, a non-transitory machine-readable storagedevice may store a set of instructions that, when executed by at leastone processor, causes the at least one processor to perform the methodoperations discussed within the present disclosure.

FIG. 1 is a network diagram illustrating a network environment 100suitable for operating a dental device 106, according to some exampleembodiments. The network environment 100 includes an imaging system 114and a server 110, communicatively coupled to each other via a network104. The imaging system 114 and the server 110 may each be implementedin a computer system, in whole or in part, as described below withrespect to FIG. 22.

The server 110 may be part of a network-based system. For example, thenetwork-based system may be or include a cloud-based server system thatprovides additional information, such as three-dimensional models ofspecimens, to the imaging system 114.

FIG. 1 illustrates a user 102 using the imaging system 114. The user 102may be a human user (e.g., a human being), a machine user (e.g., acomputer configured by a software program to interact with the dentaldevice 106), or any suitable combination thereof (e.g., a human assistedby a machine or a machine supervised by a human). The user 102 is notpart of the network environment 100, but is associated with the imagingsystem 114 and may be a user 102 of the imaging system 114.

The imaging system 114 includes a dental device 106 and a localcomputing device 112. The dental device 106 may include a dentalinstrument such as a dental handpiece, scalpel, syringe.

The local computing device 112 may be a computing device with a displaysuch as a smartphone, a tablet computer, or a laptop computer. The user102 may be a user of an application in the local computing device 112.The application may include an imaging application configured to detecta region of interest (e.g., gum disease) at the physical object 108 andprovide a visualization of the region of interest to the user 102. Inone example embodiment, the physical object 108 includes amouth/gum/teeth of a patient 118. The reference object 116 istemporarily coupled to the mouth of the patient 118. For example, thereference object 116 includes a custom-bite block that the patient 118bites.

The dental device 106 is capable of tracking its relative position andorientation in space relative to the reference object 116. For example,the dental device 106 includes optical sensors (e.g., depth-enabled 3Dcamera, image camera), inertial sensors (e.g., gyroscope,accelerometer), wireless sensors (Bluetooth, Wi-Fi), and GPS sensor, todetermine the location of the dental device 106 within a real worldenvironment. In another example, the location, position, and orientationof the dental device 106 is determined relative to the reference object116 (e.g., an object that is coupled and remains temporarily fixed tothe teeth of a patient).

Any of the machines, databases, or devices shown in FIG. 1 may beimplemented in a general-purpose computer modified (e.g., configured orprogrammed) by software to be a special-purpose computer to perform oneor more of the functions described herein for that machine, database, ordevice. For example, a computer system able to implement any one or moreof the methodologies described herein is discussed below with respect toFIG. 6 to FIG. 8. As used herein, a “database” is a data storageresource and may store data structured as a text file, a table, aspreadsheet, a relational database (e.g., an object-relationaldatabase), a triple store, a hierarchical data store, or any suitablecombination thereof. Moreover, any two or more of the machines,databases, or devices illustrated in FIG. 1 may be combined into asingle machine, and the functions described herein for any singlemachine, database, or device may be subdivided among multiple machines,databases, or devices.

The network 104 may be any network that enables communication between oramong machines (e.g., server 110), databases, and devices (e.g., dentaldevice 106). Accordingly, the network 104 may be a wired network, awireless network (e.g., a mobile or cellular network), or any suitablecombination thereof. The network 104 may include one or more portionsthat constitute a private network, a public network (e.g., theInternet), or any suitable combination thereof.

FIG. 2 is a block diagram illustrating modules (e.g., components) of theimaging system 114, according to some example embodiments. The imagingsystem 114 includes sensors 202, a display 204, a storage device 206, aprocessor 208, a composite image application 210, a mixed realityapplication 216, an optical sensor 212, and an inertial sensor 214.

The processor 208 includes the composite image application 210 and themixed reality application 216. The composite image application 210generates a composite image based on different image sources (e.g.,intraoral and CBCT). The composite image includes a visual indication ofthe region of the interest (e.g., gingival surface). The mixed realityapplication 216 generates augmented information based on the location ofthe dental device 106 and the three-dimensional model of the teeth andgum of the patient.

The mixed reality application 216 merges information from the real andvirtual world to produce new environments and visualizations, wherephysical and digital objects co-exist and interact in real-time. Mixedreality is a hybrid of augmented reality (AR) and virtual reality (VR).In one example, the mixed reality application 216 includes a combinationof AR and VR aspects.

In one example, the mixed reality application 216 includes an ARapplication that allows the user 102 to experience information, such asin the form of a three-dimensional (or two-dimensional) virtual objectoverlaid on an image of the physical object 108 captured by a camera ofthe imaging system 114. The physical object 108 may include a visualreference that the AR application can identify. A visualization of theadditional information, such as the three-dimensional virtual objectoverlaid or engaged with an image of the physical object 108, isgenerated in a display of the imaging system 114. The three-dimensionalvirtual object may be selected based on the recognized visual reference(e.g., reference object 116) or captured image of the physical object108. A rendering of the visualization of the three-dimensional virtualobject may be based on a position of the display relative to thereference object 116. Other augmented reality applications allow a userto experience visualization of the additional information overlaid ontop of a view or an image of any object in the real physical world. Thevirtual object may include a three-dimensional virtual object, atwo-dimensional virtual object. An image of the virtual object may berendered at the imaging system 114.

In one example, a system and method for creating virtual content using ahead-mounted device is described. The head-mounted device can be used tocreate virtual content without using a client device (e.g., laptop ordesktop). The wearer of the head-mounted device determines virtualcontent to be associated with the physical object 108. The head-mounteddevice then associates the virtual user interface with identifiers ofthe physical object 108 and tracking data related to the physical object108. The virtual user interface is displayed in relation to the image ofthe physical object 108.

In another example, the mixed reality application 216 displays theaugmented information in a display screen of the local computing device112.

In one example embodiment, the imaging system 114 may communicate overthe network 104 with the server 110 to retrieve a portion of a databaseof visual references (e.g., images from different specimens).

Any one or more of the modules described herein may be implemented usinghardware (e.g., a processor of a machine) or a combination of hardwareand software. For example, any module described herein may configure aprocessor to perform the operations described herein for that module.Moreover, any two or more of these modules may be combined into a singlemodule, and the functions described herein for a single module may besubdivided among multiple modules. Furthermore, according to variousexample embodiments, modules described herein as being implementedwithin a single machine, database, or device may be distributed acrossmultiple machines, databases, or devices.

FIG. 3 illustrates a composite image application in accordance with oneexample embodiment. The composite image application 210 generates acomposite image based on different image sources (e.g., intraoral andCBCT). The composite image includes a visual indication of the region ofthe interest (e.g., gingival surface). In one example, the compositeimage application 210 comprises an intraoral scanner image module 302, acone beam computerized tomography (CBCT) image module 304, a compositeimage module 306, and a gingival surface detection module 308.

The intraoral scanner image module 302 communicates with an intraoralscanner and accesses first image data of the patient 118 from theintraoral scanner. Examples of intraoral scanner include, but are notlimited to, light projection and capture, laser confocal, AWS (ActiveWavefront Sampling), and stereo-photogrammetry.

The intraoral scanner image module 302 includes a gingival surfacedetection module 308 that detects gingival surface based on the firstimage data. For example, the gingival surface detection module 308determines depth of tissue based on the image data and compares thedepth of tissue to a predefined lookup table of gingival depth.

The cone beam computerized tomography (CBCT) image module 304communicates with a cone beam computerized tomography (CBCT) andaccesses second image data of the patient 118 from the CBCT.

The composite image module 306 generates a composite image (of thepatient 118's teeth/gum) based on the first image data from theintraoral scanner image module 302 and the second image data from thecone beam computerized tomography (CBCT) image module 304. In oneexample, the composite image module 306 uses image segmentation, imageregistration/alignment of images to generate the composite image. Forexample, the composite image module 306 identifies a common region ofthe specimen in the first imaging data and the second imaging data andaligns the first imaging data with the second imaging data based on theidentified common region. The composite image module 306 registering thecomposite image when the common regions of the specimen are aligned inthe imaging data.

FIG. 4 illustrates an augmented reality (AR) application in accordancewith one example embodiment. The mixed reality application 216 generatesaugmented information based on the location of the dental device 106 andthe three-dimensional model of the teeth and gum of the patient 118.

In one example, mixed reality application 216 comprises dental diseaseMR display module 402 and dental instrument MR display module 404. Thedental disease MR display module 402 indicates a region of interest in adisplay of the 3D model. For example, the dental disease MR displaymodule 402 indicates a tooth decay area in a display of the 3D model, asuggested shape for a root canal in a display of the 3D model, regionsof the tooth for removal for a crown procedure in a display of the 3Dmodel, or a bone region of a projected injection site in a display ofthe 3D model.

The dental instrument MR display module 404 displays a virtualrepresentation of the dental device 106 relative to the 3D model of themouth of the patient 118 (e.g., teeth and gum of the patient 118). Inone example, the location of the dental device 106 is determinedrelative to the reference object 116 based on the sensors in dentaldevice 106 and reference object 116.

In other example embodiment, the mixed reality application 216 can beused for medical and surgical procedures to display in real-time animage of a surgical instrument operated by a medical professional inrelation to digital information that indicate an area of interest on areal-time image of a body part of the patient.

FIG. 5 illustrates a dental disease AR display module in accordance withone example embodiment. The dental instrument MR display module 404includes a frame of reference module 502, and an MR guidance module 504.The frame of reference module 502 determines a location of the sensorsin the dental device 106 relative to the sensors in the reference object116. In one example, the sensors include a combination of a gyroscope,an accelerometer, inertial sensor 214, wireless communication device(e.g., Bluetooth, WiFi), or any other sensor that detects a position,location, orientation of the sensors in the dental device 106 relativeto the sensors in the reference object 116. In another example, thesensors in the dental device 106 communicate with the sensors in thereference object 116. In yet another example, the sensors in the dentaldevice 106 and the sensors in the reference object 116 both communicatewith the frame of reference module 502. In yet another example, theframe of reference module 502 uses the optical sensor 212 to detect aposition, location, orientation of the sensors in the dental device 106.For example, the frame of reference module 502 detects a first uniquevisual marker of the dental instrument and a second unique visual markerof the reference object 116. The frame of reference module 502 can thusdetermine a relative distance, position, and orientation of the dentaldevice 106 relative to the reference object 116.

It is noted that the sensors of the dental device 106 are positioned ata predefined location on the dental device 106. For example, thedistance between a tip of the dental instrument and the sensors in thedental device 106 are predefined. In one example, the sensors may becoupled to any portion of the dental device 106. A lookup table thatdefines the relative distances may be updated based on the measureddistances between the sensors and other portions of the dental device106.

In one example, the reference object 116 is at a predetermined locationrelative to the teeth of the patient. Sensors in the reference object116 are located at a predefined distance related to the reference object116. For example, the distance between an end of the reference object116 and the sensors in the reference object 116 are predefined. In oneexample, the sensors may be coupled to any portion of the referenceobject 116. A lookup table that defines the relative distances may beupdated based on the measured distances between the sensors and otherportions of the reference object 116. In another example embodiment, thereference object 116 is custom-printed or custom-shaped based on theteeth of the patient.

The MR guidance module 504 determines a relative position, location,orientation of the dental device 106 relative to the reference object116. In another example, the MR guidance module 504 determines therelative distance between the dental device 106 and the reference object116.

The MR guidance module 504 accesses a 3D model (or a composite image) ofthe teeth/gum of the patient. The MR guidance module 504 initializes andcalibrates the location of the reference object 116 relative to theteeth of the patient based on the predefined distance/location betweenthe reference object 116 relative to the teeth of the patient, and thepredefined distance/location between the sensors of the reference object116 relative to the reference object 116.

The MR guidance module 504 determines a location of the dental device106 relative to the reference object 116 based on the detected position,location, orientation of the dental device 106 relative to the referenceobject 116.

The MR guidance module 504 causes a display of a virtual dentalinstrument (or any other type of visual indicator) relative to the 3Dmodel (or composite image) of the teeth/gum of the patient 118 based onthe position, location, orientation and distance of the sensors in thedental device 106 relative to the reference object 116. As such, the MRguidance module 504 provides real-time feedback (of the location of thedental device 106) to the user 102 (e.g., dentist) of the imaging system114.

In another example, the MR guidance module 504 causes display of aregion of interest in the 3D model or composite image based on thedental disease MR display module 402. For example, the MR guidancemodule 504 displays the location of the dental device 106 relative to ahighlighted region of interest in the 3D model or composite image. Inanother example, the MR guidance module 504 provides virtual displayindicators in display 204 to guide the user 102 on how to perform aprocedure (e.g., where to position and operate the dental device 106 onthe patient 118).

FIG. 6 illustrates a method 600 for generating a composite image inaccordance with one example embodiment. At block 602, the compositeimage application 210 accesses first sensor data from a first imagingsource (e.g., intraoral scanner). At block 604, the composite imageapplication 210 accesses second sensor data from a second imaging source(e.g., cone beam computerized tomography). At block 606, the compositeimage application 210 identifies common regions between the first sensordata and second sensor data (e.g., same parts of a tooth). At block 608,the composite image application 210 aligns the first sensor data and thesecond sensor data based on the common regions. At block 610, thecomposite image application 210 generates a composite image based on thealigned first and second sensor data.

FIG. 7 illustrates a method 700 for generating a 3D model in accordancewith one example embodiment. At block 702, the composite imageapplication 210 identifies gingival surface based on first sensor data.At block 704, the composite image application 210 generates virtualindicator for the gingival surface on the composite image. At block 706,the composite image application 210 generates a 3D model that includesthe gingival surface.

FIG. 8 illustrates a method 800 for generating AR information inaccordance with one example embodiment. At block 802, the mixed realityapplication 216 identifies a frame of reference by detecting thelocation of the reference object 116. At block 804, the mixed realityapplication 216 determines a position and orientation (pose) of aninstrument relative to the reference object 116. At block 806, the mixedreality application 216 generates mixed reality dental information(e.g., digital information that is superimposed on a live view or areal-time display of the teeth of the patient 118). The mixed realitydental information is based on the pose (e.g., location, orientation,position) of the dental device 106 relative to the 3D model. At block808, the mixed reality application 216 displays the mixed reality dentalinformation in the display 204.

In block 902, routine 900 accesses first imaging data of a specimenusing a first sensor device, the first imaging data comprisingvolumetric data. In block 904, routine 900 accesses second imaging dataof the specimen using a second sensor device, the second imaging datacomprising surface data. In block 906, routine 900 generates a compositeimage based on the first and second imaging data, the composite imageindicating the volumetric data and the surface data of the specimen. Inblock 908, routine 900 accesses first sensor data of a reference objectresting in a mouth of a patient, the reference object being at apredefined position relative to the mouth of the patient. In block 910,routine 900 accesses second sensor data of a sensor coupled to a dentalinstrument. In block 912, routine 900 determines a position of thedental instrument relative to the reference object based on the firstand second sensor data. In block 914, routine 900 displays a virtualrepresentation of the dental instrument relative to a virtualrepresentation of the mouth of the patient based on the position of thedental instrument relative to the reference object.

FIG. 10 illustrates an example process 1000 of the composite imageapplication 210 in accordance with one example embodiment. Thevolumetric data from CBCT 1002 provide surface data of tooth 1004. Thesurface data from intraoral scan 1006 is used to identify crown surfacedata 1008. At image registration 1010, both the surface data of tooth1004 and crown surface data 1008 are registered to generate a compositeimage. The composite image indicates CBCT augmented with gingivalsurface 1012 that can be used for clinical measurements 1014. Examplesof clinical measurements 1014 include pocket depth, tissue biotype, andareas of inflammation. Data from the clinical measurements 1014 can beused to generate 3D manipulative data showing dental disease 1016.

FIG. 11 illustrates an example process 1100 of the dental instrument MRdisplay module 404 in accordance with one example embodiment. In block1108, the virtual positioning of the dental device 106 relative to thepatient 118 is based on a reference object 116 (e.g., 3D printed custombite block with sensors) positioned inside the mouth of the patient 118at block 1104 and the relative position (e.g., 6 axis relative position)between the reference object 116 and the dental device 106 at block1102. The virtual positioning of the dental device 106 relative tocustom bite block inside the patient 118 is also based on the 3D datamodel showing dental disease and anatomical features at block 1106.

The geometric parameters (e.g., bur dimension, needle length) of thedental device 106 at block 1124 are used along with the virtualpositioning of the dental device 106 for clinical guidance at block1110. Examples of clinical guidance include dental injection guidance atblock 1112, surgical implant placement at block 1116, and gum surgeriesat block 1120, pulpal detection for filling preparation at block 1114,virtual crown preparation guidance (e.g., occlusal reduction) at block1118, and oral surgery applications (e.g., biopsy) at block 1122.

FIG. 12 illustrates an example 1200 of combining a first and seconddental imaging data to generate a composite image in accordance with oneexample embodiment. A common region 1208 that is common to both the softtissue image 1202 and the tooth surface image 1204 is identified. Theimaging system 114 uses the common region 1208 for registration andcombines both the soft tissue image 1202 and the tooth surface image1204 into a composite image 1206.

FIG. 13 illustrates a reference object in accordance with one exampleembodiment. The diagram 1300 illustrates a custom bite block 1304 thatis customized to a bite of the patient 118 (e.g., the custom bite block1304 is unique and based on the patient 118's bite). A micro sensor 1306is coupled (e.g., embedded at a known position relative to the custombite block 1304) to a predetermined location on the custom bite block1304. The diagram 1302 illustrates the custom bite block 1304temporarily coupled to the mouth 1308 of the patient 118. The patient118 bites on the custom bite block 1304. As such, the location of atooth in the mouth 1412 of the patient 118 can be determined relative tothe custom bite block 1304 and the micro sensor 1306.

FIG. 14 is a diagram 1400 illustrating an example operation of thecustom bite block 1404 in accordance with one example embodiment. Theenlargement 1402 illustrates the custom bite block 1404 coupled to themouth 1412 of the patient 118. The patient 118 sits on a chair 1416.

The user 102 (e.g., a dentist) operates the dental device 106 in themouth 1412 of the patient 118. The imaging system 114 includes a display1414 that displays real-time (or near real-time) view of the location ofthe dental device 106 relative to a digital representation of a decay1406 in the tooth 1408 of the patient 118. In other words, the display1414 updates the information (e.g., image of the dental device 106 andaugmented information representing the decay 1406) in real-time.

In another example embodiment, the presently described imaging system114 can be used for other surgical operations where the user 102operates on a portion of a body of the patient 118. For example, theimaging system 114 provides a visual indicator that points or highlightsa portion of the real-time image of the portion of the body.Furthermore, the imaging system 114 provides guidance (via augmentedreality information such as virtual arrows displayed on the real-timeimage) on how to operate on the portion of the body of the patient 118based on the relative location of an instrument held by the user 102 andthe portion of the body of the patient 118. The reference object 116 maybe coupled to a predefined location on the portion of the body of thepatient 118. As such, the distance and position of the instrumentrelative to the body of the patient 118 can be determined.

In one example embodiment, The MR guidance module 504 provides a visualguide to a user to visually guide a syringe to an injection siterelative to a jawbone.

FIG. 15 illustrates an example of a dental device 106 with inertialsensors in accordance with one example embodiment. Examples of dentaldevice 106 include a dental handpiece 1502, a scalpel, and a syringe. Asensor such as a micro tracker (e.g., sensor 1504, sensor 1506) isattached to the dental handpiece 1502. Exact geometries of the dentalhandpiece 1502 and attachments (e.g. dental burs) are known anddigitized.

FIG. 16 illustrates a dental instrument with inertial sensors inaccordance with one example embodiment. Examples of dental device 106include a syringe 1602. A sensor such as a microtracker (e.g., sensor1604, sensor 1606) is attached to the dental handpiece 1502. Exactgeometries of the dental handpiece 1502 and attachments (e.g. syringeneedles) are known and digitized.

FIG. 17 illustrates a cross-section 1700 of a composite image of a toothin accordance with one example embodiment. Examples of periodontalcalculations performed by the composite image application 210 include:

Probing Depth (distance between top contact between tooth/gum and crestof bone—distance between point 1702 and point 1704)

Attachment Loss (periodontitis) (approximate height difference betweenenamel/root junction and alveolar bone crest—distance between point 1708and point 1710)

Gingival Swelling (gingivitis) (probing depth+attachmentloss—physiological pocket depth (about 2-3 mm))

Gingival biotype (thick or thin) (thickness depth between gum andsupporting bone based average thickness of area 1706).

FIG. 18 illustrates updating a composite image of a cross-section 1800of a tooth in accordance with one example embodiment. The compositeimage 1802 can be updated based on additional data (e.g., updatedgeometric or colored data in a new scan of the same tooth). An updatedimage 1804 is combined to the composite image 1802 to generate anupdated composite image 1810.

Differences from the composite image 1802 and the updated image 1804 areindicated in the updated composite image 1810. For example, thedifferences include an area of surface addition 1806 and an area ofsurface wear 1808. The area of surface addition 1806 and area of surfacewear 1808 can be used to identify areas of tartar buildup andgingivitis.

FIG. 19 illustrates an example operation 1900 of the imaging system inaccordance with one example embodiment. The tooth surface 1904 and toothnerve 1902 are displayed in the display 204. The display 204 alsodisplays the decay area 1906 on the corresponding area on the toothsurface 1904. The MR guidance module 504 provides a visual guide for thedental instrument 1908 of the user 102 to the decay area 1906.

FIG. 20 illustrates another example operation 2000 of the imaging systemin accordance with one example embodiment. The tooth surface 1904 andtooth nerve 1902 are displayed in the display 204. The MR guidancemodule 504 provides a visual guide for the dental instrument 1908 to theroot canal shape 2002.

FIG. 21 illustrates another example operation 2100 of the imaging systemin accordance with one example embodiment. The MR guidance module 504provides a visual guide for the dental instrument 1908 to bur the toothshape to be removed 2102.

FIG. 22 is a diagrammatic representation of the machine 2200 withinwhich instructions 2204 (e.g., software, a program, an application, anapplet, an app, or other executable code) for causing the machine 2200to perform any one or more of the methodologies discussed herein may beexecuted. For example, the instructions 2204 may cause the machine 2200to execute any one or more of the methods described herein. Theinstructions 2204 transform the general, non-programmed machine 2200into a particular machine 2200 programmed to carry out the described andillustrated functions in the manner described. The machine 2200 mayoperate as a standalone device or may be coupled (e.g., networked) toother machines. In a networked deployment, the machine 2200 may operatein the capacity of a server machine or a client machine in aserver-client network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The machine 2200 maycomprise, but not be limited to, a server computer, a client computer, apersonal computer (PC), a tablet computer, a laptop computer, a netbook,a set-top box (STB), a PDA, an entertainment media system, a cellulartelephone, a smart phone, a mobile device, a wearable device (e.g., asmart watch), a smart home device (e.g., a smart appliance), other smartdevices, a web appliance, a network router, a network switch, a networkbridge, or any machine capable of executing the instructions 2204,sequentially or otherwise, that specify actions to be taken by themachine 2200. Further, while only a single machine 2200 is illustrated,the term “machine” shall also be taken to include a collection ofmachines that individually or jointly execute the instructions 2204 toperform any one or more of the methodologies discussed herein.

The machine 2200 may include processors 2206, memory 2208, and I/Ocomponents 2242, which may be configured to communicate with each othervia a bus 2244. In an example embodiment, the processors 2206 (e.g., aCentral Processing Unit (CPU), a Reduced Instruction Set Computing(RISC) Processor, a Complex Instruction Set Computing (CISC) Processor,a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), anASIC, a Radio-Frequency Integrated Circuit (RFIC), another Processor, orany suitable combination thereof) may include, for example, a Processor2202 and a Processor 2210 that execute the instructions 2204. The term“Processor” is intended to include multi-core processors that maycomprise two or more independent processors (sometimes referred to as“cores”) that may execute instructions contemporaneously. Although FIG.22 shows multiple processors 2206, the machine 2200 may include a singleProcessor with a single core, a single Processor with multiple cores(e.g., a multi-core Processor), multiple processors with a single core,multiple processors with multiples cores, or any combination thereof.

The memory 2208 includes a main memory 2212, a static memory 2214, and astorage unit 2216, both accessible to the processors 2206 via the bus2244. The main memory 2208, the static memory 2214, and storage unit2216 store the instructions 2204 embodying any one or more of themethodologies or functions described herein. The instructions 2204 mayalso reside, completely or partially, within the main memory 2212,within the static memory 2214, within machine-readable medium 2218within the storage unit 2216, within at least one of the processors 2206(e.g., within the Processor's cache memory), or any suitable combinationthereof, during execution thereof by the machine 2200.

The I/O components 2242 may include a wide variety of components toreceive input, provide output, produce output, transmit information,exchange information, capture measurements, and so on. The specific I/Ocomponents 2242 that are included in a particular machine will depend onthe type of machine. For example, portable machines such as mobilephones may include a touch input device or other such input mechanisms,while a headless server machine will likely not include such a touchinput device. It will be appreciated that the I/O components 2242 mayinclude many other components that are not shown in FIG. 22. In variousexample embodiments, the I/O components 2242 may include outputcomponents 2228 and input components 2230. The output components 2228may include visual components (e.g., a display such as a plasma displaypanel (PDP), a light emitting diode (LED) display, a liquid crystaldisplay (LCD), a projector, or a cathode ray tube (CRT)), acousticcomponents (e.g., speakers), haptic components (e.g., a vibratory motor,resistance mechanisms), other signal generators, and so forth. The inputcomponents 2230 may include alphanumeric input components (e.g., akeyboard, a touch screen configured to receive alphanumeric input, aphoto-optical keyboard, or other alphanumeric input components),point-based input components (e.g., a mouse, a touchpad, a trackball, ajoystick, a motion sensor, or another pointing instrument), tactileinput components (e.g., a physical button, a touch screen that provideslocation and/or force of touches or touch gestures, or other tactileinput components), audio input components (e.g., a microphone), and thelike.

In further example embodiments, the I/O components 2242 may includebiometric components 2232, motion components 2234, environmentalcomponents 2236, or position components 2238, among a wide array ofother components. For example, the biometric components 2232 includecomponents to detect expressions (e.g., hand expressions, facialexpressions, vocal expressions, body gestures, or eye tracking), measurebiosignals (e.g., blood pressure, heart rate, body temperature,perspiration, or brain waves), identify a person (e.g., voiceidentification, retinal identification, facial identification,fingerprint identification, or electroencephalogram-basedidentification), and the like. The motion components 2234 includeacceleration sensor components (e.g., accelerometer), gravitation sensorcomponents, rotation sensor components (e.g., gyroscope), and so forth.The environmental components 2236 include, for example, illuminationsensor components (e.g., photometer), temperature sensor components(e.g., one or more thermometers that detect ambient temperature),humidity sensor components, pressure sensor components (e.g.,barometer), acoustic sensor components (e.g., one or more microphonesthat detect background noise), proximity sensor components (e.g.,infrared sensors that detect nearby objects), gas sensors (e.g., gasdetection sensors to detection concentrations of hazardous gases forsafety or to measure pollutants in the atmosphere), or other componentsthat may provide indications, measurements, or signals corresponding toa surrounding physical environment. The position components 2238 includelocation sensor components (e.g., a GPS receiver Component), altitudesensor components (e.g., altimeters or barometers that detect airpressure from which altitude may be derived), orientation sensorcomponents (e.g., magnetometers), and the like.

Communication may be implemented using a wide variety of technologies.The I/O components 2242 further include communication components 2240operable to couple the machine 2200 to a network 2220 or devices 2222via a coupling 2224 and a coupling 2226, respectively. For example, thecommunication components 2240 may include a network interface Componentor another suitable device to interface with the network 2220. Infurther examples, the communication components 2240 may include wiredcommunication components, wireless communication components, cellularcommunication components, Near Field Communication (NFC) components,Bluetooth® components (e.g., Bluetooth® Low Energy), WiFi® components,and other communication components to provide communication via othermodalities. The devices 2222 may be another machine or any of a widevariety of peripheral devices (e.g., a peripheral device coupled via aUSB).

Moreover, the communication components 2240 may detect identifiers orinclude components operable to detect identifiers. For example, thecommunication components 2240 may include Radio Frequency Identification(RFID) tag reader components, NFC smart tag detection components,optical reader components (e.g., an optical sensor to detectone-dimensional bar codes such as Universal Product Code (UPC) bar code,multi-dimensional bar codes such as Quick Response (QR) code, Azteccode, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2Dbar code, and other optical codes), or acoustic detection components(e.g., microphones to identify tagged audio signals). In addition, avariety of information may be derived via the communication components2240, such as location via Internet Protocol (IP) geolocation, locationvia Wi-Fi® signal triangulation, location via detecting an NFC beaconsignal that may indicate a particular location, and so forth.

The various memories (e.g., memory 2208, main memory 2212, static memory2214, and/or memory of the processors 2206) and/or storage unit 2216 maystore one or more sets of instructions and data structures (e.g.,software) embodying or used by any one or more of the methodologies orfunctions described herein. These instructions (e.g., the instructions2204), when executed by processors 2206, cause various operations toimplement the disclosed embodiments.

The instructions 2204 may be transmitted or received over the network2220, using a transmission medium, via a network interface device (e.g.,a network interface Component included in the communication components2240) and using any one of a number of well-known transfer protocols(e.g., hypertext transfer protocol (HTTP)). Similarly, the instructions2204 may be transmitted or received using a transmission medium via thecoupling 2226 (e.g., a peer-to-peer coupling) to the devices 2222.

Although an embodiment has been described with reference to specificexample embodiments, it will be evident that various modifications andchanges may be made to these embodiments without departing from thebroader scope of the present disclosure. Accordingly, the specificationand drawings are to be regarded in an illustrative rather than arestrictive sense. The accompanying drawings that form a part hereof,show by way of illustration, and not of limitation, specific embodimentsin which the subject matter may be practiced. The embodimentsillustrated are described in sufficient detail to enable those skilledin the art to practice the teachings disclosed herein. Other embodimentsmay be utilized and derived therefrom, such that structural and logicalsubstitutions and changes may be made without departing from the scopeof this disclosure. This Detailed Description, therefore, is not to betaken in a limiting sense, and the scope of various embodiments isdefined only by the appended claims, along with the full range ofequivalents to which such claims are entitled.

Such embodiments of the inventive subject matter may be referred toherein, individually and/or collectively, by the term “invention” merelyfor convenience and without intending to voluntarily limit the scope ofthis application to any single invention or inventive concept if morethan one is in fact disclosed. Thus, although specific embodiments havebeen illustrated and described herein, it should be appreciated that anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

The Abstract of the Disclosure is provided to allow the reader toquickly ascertain the nature of the technical disclosure. It issubmitted with the understanding that it will not be used to interpretor limit the scope or meaning of the claims. In addition, in theforegoing Detailed Description, it can be seen that various features aregrouped together in a single embodiment for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter lies in less than allfeatures of a single disclosed embodiment. Thus, the following claimsare hereby incorporated into the Detailed Description, with each claimstanding on its own as a separate embodiment.

EXAMPLES

Example 1 includes a method comprising: accessing first imaging data ofa specimen using a first sensor device, the first imaging datacomprising volumetric data; accessing second imaging data of thespecimen using a second sensor device, the second imaging datacomprising surface data; registering a common anatomical region of thespecimen in the first imaging data and the second imaging data; andgenerating a composite image based on the registered common anatomicalregion, the composite image indicating the volumetric data and thesurface data of the specimen.

Example 2 includes example 1, further comprising: determining clinicalmeasurements of the specimen based on the first and second imaging data;generating a three-dimensional model of the specimen based on theclinical measurements and the composite image; and measuring dentalperiodontal health with the three-dimensional model of the specimen.

Example 3 includes any of the above examples, wherein the clinicalmeasurements indicate at least one of a pocket depth, a tissue biotype,areas of inflammation and tissue damage, exposure of dental furcation,or dental attachment loss.

Example 4 includes any of the above examples, wherein registering thecommon anatomical region further comprises: identifying the commonanatomical region of the specimen in the first imaging data and thesecond imaging data; and aligning the first imaging data with the secondimaging data based on the identified common anatomical region.

Example 5 includes any of the above examples, wherein the first sensortool comprises a cone beam CT scan, the first imaging data indicatingbone volume of the specimen, wherein the second sensor tool comprises anintraoral scan, the volumetric data comprising coloring attributescorresponding to tissue quality of the specimen.

Example 6 includes any of the above examples, further comprising:accessing first sensor data of a reference object resting in a mouth ofa patient, the reference object being at a predefined position relativeto the mouth of the patient; accessing second sensor data of a sensorcoupled to a dental instrument; determining a position of the dentalinstrument relative to the reference object based on the first andsecond sensor data; and displaying a virtual representation of thedental instrument relative to a virtual representation of the mouth ofthe patient based on the position of the dental instrument relative tothe reference object.

Example 7 includes any of the above examples, further comprising:providing a patient with a bite block that has been customized to fitthe patient, the bite block configured to be temporarily locked with theupper and lower jaw of the patient, the bite block forming a predefinedframe of reference based on the position of the bite block relative to atooth of the patient; accessing position data from a position sensorattached to a dental instrument, the position data being relative to thebite block; computing a relative position between the dental instrumentand a region of interest of the patient; and displaying, a virtualposition of the dental instrument relative to a 3D model of a mouth ofthe patient in real time, based on the relative position between thedental instrument and the region of interest, wherein the 3D model isbased on the composite image.

Example 8 includes any of the above examples, further comprising:indicating a tooth decay area and a dental nerve area in a display ofthe 3D model.

Example 9 includes any of the above examples, further comprising:indicating a suggested shape for a root canal in a display of the 3Dmodel.

Example 10 includes any of the above examples, further comprising:indicating a dental nerve area and a tooth removal area for a crownprocedure in a display of the 3D model.

Example 11 includes any of the above examples, further comprising:indicating a bone region of a projected injection site in a display ofthe 3D model.

Example 12 includes any of the above examples, wherein the secondimaging data is based on at least one of a fluorescent image withexcitation wavelength of about 270 nm to about 370 nm, emissionwavelength of about 305 nm to about 500 nm, near-infrared imaging thatindicates reflectance and transmission, or optical coherence tomographyin the near infrared spectrum.

Example 13 includes any of the above examples, further comprising:accessing updated imaging data of the specimen using the first or secondsensor device; updating the composite image based on the third imagingdata; and identifying a region of difference based on the updatedcomposite image and the composite image, the region of differenceindicating an area of surface wear on a tooth or an area of surfaceaddition on the tooth.

Example 14 includes any of the above examples, further comprising:identifying a region of interest in the composite image based on ananalysis of the first or second imaging data; generating a virtualindicator that indicates the region of interest; causing a display ofthe virtual indicator in the composite image or an image of thespecimen; and operating the dental instrument with a robotic device thatis configured to operate on the specimen at the region of interest.

Example 15 includes a computing apparatus, the computing apparatuscomprising: a Processor; and a memory storing instructions that, whenexecuted by the Processor, configure the apparatus to: access firstimaging data of a specimen using a first sensor device, the firstimaging data comprising volumetric data; access second imaging data ofthe specimen using a second sensor device, the second imaging datacomprising surface data; and generate a composite image based on thefirst and second imaging data, the composite image indicating thevolumetric data and the surface data of the specimen.

1. A method comprising: accessing first imaging data of a specimen usinga first sensor device, the first imaging data comprising volumetricdata; accessing second imaging data of the specimen using a secondsensor device, the second imaging data comprising surface data;registering a common anatomical region of the specimen in the firstimaging data and the second imaging data; and generating a compositeimage based on the registered common anatomical region, the compositeimage indicating the volumetric data and the surface data of thespecimen, wherein the first sensor tool comprises a cone beam CT scan,the first imaging data indicating bone volume of the specimen, whereinthe second sensor tool comprises an intraoral scan.
 2. The method ofclaim 1, further comprising: determining clinical measurements of thespecimen based on the first and second imaging data; generating athree-dimensional model of the specimen based on the clinicalmeasurements and the composite image; and measuring dental periodontalhealth with the three-dimensional model of the specimen.
 3. The methodof claim 2, wherein the clinical measurements indicate at least one of apocket depth, a tissue biotype, areas of inflammation and tissue damage,exposure of dental furcation, or dental attachment loss.
 4. The methodof claim 1, wherein registering the common anatomical region furthercomprises: identifying the common anatomical region of the specimen inthe first imaging data and the second imaging data; and aligning thefirst imaging data with the second imaging data based on the identifiedcommon anatomical region.
 5. The method of claim 1, wherein thevolumetric data comprising coloring attributes corresponding to tissuequality of the specimen.
 6. The method of claim 1, further comprising:accessing first sensor data of a reference object resting in a mouth ofa patient, the reference object being at a predefined position relativeto the mouth of the patient; accessing second sensor data of a sensorcoupled to a dental instrument; determining a position of the dentalinstrument relative to the reference object based on the first andsecond sensor data; and displaying a virtual representation of thedental instrument relative to a virtual representation of the mouth ofthe patient based on the position of the dental instrument relative tothe reference object.
 7. The method of claim 1, further comprising:providing a patient with a bite block that has been customized to fitthe patient, the bite block configured to be temporarily locked with theupper and lower jaw of the patient, the bite block forming a predefinedframe of reference based on the position of the bite block relative to atooth of the patient; accessing position data from a position sensorattached to a dental instrument, the position data being relative to thebite block; computing a relative position between the dental instrumentand a region of interest of the patient; and displaying, a virtualposition of the dental instrument relative to a 3D model of a mouth ofthe patient in real time, based on the relative position between thedental instrument and the region of interest, wherein the 3D model isbased on the composite image.
 8. The method of claim 7, furthercomprising: indicating a tooth decay area and a dental nerve area in adisplay of the 3D model.
 9. The method of claim 7, further comprising:indicating a suggested shape for a root canal in a display of the 3Dmodel.
 10. The method of claim 7, further comprising: indicating adental nerve area and a tooth removal area for a crown procedure in adisplay of the 3D model.
 11. The method of claim 7, further comprising:indicating a bone region of a projected injection site in a display ofthe 3D model.
 12. The method of claim 1, wherein the second imaging datais based on at least one of a fluorescent image with excitationwavelength of about 270 nm to about 370 nm, emission wavelength of about305 nm to about 500 nm, near-infrared imaging that indicates reflectanceand transmission, or optical coherence tomography in the near infraredspectrum.
 13. The method of claim 1, further comprising: accessingupdated imaging data of the specimen using the first or second sensordevice; updating the composite image based on the third imaging data;and identifying a region of difference based on the updated compositeimage and the composite image, the region of difference indicating anarea of surface wear on a tooth or an area of surface addition on thetooth.
 14. The method of claim 6, further comprising: identifying aregion of interest in the composite image based on an analysis of thefirst or second imaging data; generating a virtual indicator thatindicates the region of interest; causing a display of the virtualindicator in the composite image or an image of the specimen; andoperating the dental instrument with a robotic device that is configuredto operate on the specimen at the region of interest.
 15. A computingapparatus, the computing apparatus comprising: a processor; and a memorystoring instructions that, when executed by the processor, configure theapparatus to perform operations comprising: access first imaging data ofa specimen using a first sensor device, the first imaging datacomprising volumetric data; access second imaging data of the specimenusing a second sensor device, the second imaging data comprising surfacedata; register a common anatomical region of the specimen in the firstimaging data and the second imaging data; and generate a composite imagebased on the registered common anatomical region, the composite imageindicating the volumetric data and the surface data of the specimen,wherein the first sensor tool comprises a cone beam CT scan, the firstimaging data indicating bone volume of the specimen, wherein the secondsensor tool comprises an intraoral scan.
 16. The computing apparatus ofclaim 15, wherein the instructions further configure the apparatus to:determine clinical measurements of the specimen based on the first andsecond imaging data; generate a three-dimensional model of the specimenbased on the clinical measurements and the composite image; and measuredental periodontal health with the three-dimensional model of thespecimen.
 17. The computing apparatus of claim 16, wherein the clinicalmeasurements indicate at least one of a pocket depth, a tissue biotype,areas of inflammation and tissue damage, exposure of dental furcation,or dental attachment loss, wherein the volumetric data comprisingcoloring attributes corresponding to tissue quality of the specimen. 18.The computing apparatus of claim 15, wherein the instructions furtherconfigure the apparatus to: access first sensor data of a referenceobject resting in a mouth of a patient, the reference object being at apredefined position relative to the mouth of the patient; access secondsensor data of a sensor coupled to a dental instrument; determine aposition of the dental instrument relative to the reference object basedon the first and second sensor data; and display a virtualrepresentation of the dental instrument relative to a virtualrepresentation of the mouth of the patient based on the position of thedental instrument relative to the reference object.
 19. The computingapparatus of claim 15, wherein the instructions further configure theapparatus to: provide a patient with a bite block that has beencustomized to fit the patient, the bite block configured to betemporarily locked with the upper and lower jaw of the patient, the biteblock forming a predefined frame of reference based on the position ofthe bite block relative to a tooth of the patient; access position datafrom a position sensor attached to a dental instrument, the positiondata being relative to the bite block; compute a relative positionbetween the dental instrument and a region of interest of the patient;and display, a virtual position of the dental instrument relative to a3D model of a mouth of the patient in real time, based on the relativeposition between the dental instrument and the region of interest,wherein the 3D model is based on the composite image.
 20. Acomputer-readable storage medium, the computer-readable storage mediumincluding instructions that when executed by a computer, cause thecomputer to perform operations comprising: access first imaging data ofa specimen using a first sensor device, the first imaging datacomprising volumetric data; access second imaging data of the specimenusing a second sensor device, the second imaging data comprising surfacedata; register a common anatomical region of the specimen in the firstimaging data and the second imaging data; and generate a composite imagebased on the registered common anatomical region, the composite imageindicating the volumetric data and the surface data of the specimen,wherein the first sensor tool comprises a cone beam CT scan, the firstimaging data indicating bone volume of the specimen, wherein the secondsensor tool comprises an intraoral scan.